The Art of Data Augmentation

The term data augmentation refers to methods for constructing iterative optimization or sampling algorithms via the introductionof unobserved data or latent variables. For deterministic algorithms, the method was popularized in the general statistical community by the seminal article by Dempster, Laird, and Rubin on the EM algorithm for maximizing a likelihood function or, more generally, a posterior density. For stochastic algorithms, the method was popularized in the statistical literature by Tanner and Wong’s Data Augmentation algorithmfor posterior sampling and in the physics literatureby Swendsen andWang’s algorithm for sampling from the Ising and Potts models and their generalizations; in the physics literature, the method of data augmentationis referred to as the method of auxiliary variables. Data augmentationschemes were used by Tanner and Wong to make simulation feasible and simple, while auxiliary variables were adopted by Swendsen and Wang to improve the speedof iterativesimulation.In general,however, constructingdata augmentation schemes that result in both simple and fast algorithms is a matter of art in that successful strategiesvary greatlywith the (observed-data)models being considered.After an overview of data augmentation/auxiliaryvariablesand some recent developmentsinmethods for constructingsuchefŽ cientdataaugmentationschemes,we introduceaneffectivesearchstrategy that combines the ideas of marginal augmentationand conditionalaugmentation, together with a deterministic approximationmethod for selecting good augmentation schemes. We then apply this strategy to three common classes of models (speciŽ cally, multivariate t, probit regression,and mixed-effectsmodels) to obtain efŽ cient Markov chainMonte Carlo algorithms for posterior sampling. We provide theoretical and empirical evidence that the resulting algorithms, while requiring similar programming effort, can show dramatic improvement over the Gibbs samplers commonly used for these models in practice. A key featureof all thesenewalgorithmsis that they arepositiverecurrentsubchainsof nonpositive recurrentMarkov chains constructed in larger spaces.